Field of the Invention
The present invention relates to an image processing apparatus, an image capturing apparatus, an image processing method, and a non-transitory computer-readable storage medium.
Description of the Related Art
Along with the digitization of information, there have been proposals for various methods of correction processing performed on a shot image by handling the image as signal values. When capturing a subject with a digital camera and converting the subject to an image, the obtained image is considerably degraded particularly by aberration of an imaging optical system.
A blur component of an image is caused by spherical aberration, comatic aberration, image plane curvature, astigmatic aberration, and the like of the optical system. The blur component of the image due to these forms of aberration indicates that an image is formed in which, although luminous flux emitted from one point of the subject should be again gathered at one point on the imaging plane in a case where the image is aplanatic and there is no effect of diffraction, this luminous flux has a spread. Optically, this is called a point spread function (PSF), and is referred to as the blur component in an image. Image blur, for example, also refers to an image that is out of focus, but here, image blur can refer to blurring due to the effect of aberration in the above optical system even if the image is in focus. Also, color bleeding in a color image that is caused by color aberration on the axis of the optical system, color spherical aberration, and color comatic aberration, can also refer to differences in blurring between each wavelength of light. Also, color shift in the lateral direction caused by magnification color aberration of the optical system can refer to positional shift or phase shift due to differences in capture magnification between each wavelength of light.
An optical transfer function (OTF) obtained by performing a Fourier transform on the PSF is frequency component information of aberration, and is expressed as a complex number. The absolute value of the OTF, i.e., an amplitude component, is referred to as a Modulation Transfer Function (MTF), and a phase component is referred to as a PTF (Phase Transfer Function). Thus, the MTF and the PTF respectively are frequency characteristics of the amplitude component and the phase component of image degradation due to aberration. Here, the phase component is expressed as a phase angle in the below expression. Re(OTF) and Im(OTF) respectively indicate a real portion and an imaginary portion of the OTF.PTF=tan−1(Im(OTF)/Re(OTF))  (Exp. 1)
Thus, the OTF of the imaging optical system contributes degradation to the amplitude component and the phase component of the image, so in the degraded image, points of the subject are in an asymmetrically blurred state with comatic aberration. As a method for correcting degradation of amplitude (MTF) and degradation of phase (PTF), it is known to perform correction using information of the OTF of the imaging optical system. This method is referred to by the terms image recovery or image restoration, and hereinafter, processing to correct image degradation using information of the OTF of the imaging optical system is referred to as image recovery processing or recovery processing.
Following is a summary of image recovery processing. In the expression below, g(x, y) represents a degraded image, f(x, y) represents an original image, and h(x, y) represents a PSF obtained by performing an inverse Fourier transform on the above optical transfer function. Note also that * indicates convolution, and (x, y) indicates coordinates on an image.g(x,y)=h(x,y)*f(x,y)  (Exp. 2)
Also, when a Fourier transform is performed on this expression to convert to a format for display in the frequency plane, a format is obtained that is a product of each frequency, as in the below expression. H is obtained by performing a Fourier transform on the PSF, and therefore is an OTF. Coordinates in a two-dimensional frequency plane are indicated by (u, v), that is, (u, v) indicates frequency.G(u,v)=H(u,v)·F(u,v)  (Exp. 3)
In order to obtain an original image from a degraded image that has been shot, both sides may be divided by H as shown below.G(u,v)/H(u,v)=F(u,v)  (Exp. 4)
By performing an inverse Fourier transform on this F(u, v) to return to an actual plane, an original image f(x, y) is obtained as a recovered image.
Here, if R represents the result obtained by performing an inverse Fourier transform on 1/H in the above expression, it is similarly possible to obtain an original image by performing convolution processing on an image in the actual plane as shown in the below expression.g(x,y)*R(x,y)=f(x,y)  (Exp. 5)
This R(x, y) is called an image recovery filter. Because there is a noise component in the actual image, when an image recovery filter created with a perfect reciprocal of the OTF as described above is used, the noise component is amplified together with the degraded image so usually a good image is not obtained. In this regard, a method is known in which, for example as with a Wiener filter, the recovery rate of the high frequency side of the image is suppressed according to an intensity ratio of the image signal and the noise signal. As a method for correcting degradation of a color bleeding component of the image, for example, correction is considered performed when there is a uniform amount of blur for each color component of the image by the above blur component correction. Here, because the OTF varies according to shooting states such as a zoom position state or the state of an aperture diameter, it is necessary that the image recovery filter used in image recovery processing also varies according to the shooting state.
Research into this sort of image recovery technology has previously been performed, and in Japanese Patent Laid-Open No. 2006-238032, image recovery processing is disclosed in which image recovery processing is performed with a minutely small spread being set for the PSF after image recovery.
As described above, image quality can be improved by correcting aberration by performing image recovery processing on an input image that was captured. Due to the image recovery correcting lens aberration, perceived resolution of the image improves, but edge emphasis is widely known as other processing that improves perceived resolution. Edge emphasis is processing that emphasizes outlines of an image, thus improving the perceived resolution of the image.
Perceived resolution can be improved by edge emphasis, but there are cases where over-correction in image recovery is emphasized. Also, there are cases where a user can set the intensity of a sharpness function of a digital camera utilizing edge emphasis processing, but operability of the camera is considerably decreased when settings values are adjusted according to whether or not image recovery is applied. With regard to this point, above-described Japanese Patent Laid-Open No. 2006-238032 does not consider edge emphasis processing when performing image recovery.